Techniques for predicting perceptual video quality

ABSTRACT

In one embodiment of the present invention, a quality trainer and quality calculator collaborate to establish a consistent perceptual quality metric via machine learning. In a training phase, the quality trainer leverages machine intelligence techniques to create a perceptual quality model that combines objective metrics to optimally track a subjective metric assigned during viewings of training videos. Subsequently, the quality calculator applies the perceptual quality model to values for the objective metrics for a target video, thereby generating a perceptual quality score for the target video. In this fashion, the perceptual quality model judiciously fuses the objective metrics for the target video based on the visual feedback processed during the training phase. Since the contribution of each objective metric to the perceptual quality score is determined based on empirical data, the perceptual quality score is a more accurate assessment of observed video quality than conventional objective metrics.

BACKGROUND OF THE INVENTION

Field of the Invention

Embodiments of the present invention relate generally to computerscience and, more specifically, to techniques for predicting perceptualvideo quality.

Description of the Related Art

Efficiently and accurately encoding source video is essential forreal-time delivery of video content. After the encoded video content isreceived, the source video is decoded and viewed or otherwise operatedupon. Some encoding processes employ lossless compression algorithms,such as Huffman coding, to enable exact replication of the source. Bycontrast, to increase compression rates and/or reduce the size of theencoded video content, other encoding processes leverage lossy datacompression techniques that eliminate selected information, typicallyenabling only approximate reconstruction of the source. Furtherdistortion may be introduced during resizing operations in which thevideo is scaled-up to a larger resolution to match the dimensions of adisplay device.

Manually verifying the quality of delivered video is prohibitively timeconsuming. Consequently, to ensure an acceptable video watchingexperience, efficiently and accurately predicting the quality ofdelivered video is desirable. Accordingly, automated video qualityassessment is often an integral part of the encoding and streaminginfrastructure—employed in a variety of processes such as evaluatingencoders and fine-tune streaming bitrates to maintain video quality.

In one approach to assessing the quality of encoded videos, afull-reference quality metric, such as peak signal-to-noise ratio(PSNR), is used to compare the source video to the encoded video.However, while such metrics accurately reflect signal fidelity (i.e.,the faithfulness of the encoded video to the source video), thesemetrics do not reliably predict human perception of quality. Forexample, fidelity measurements typically do not reflect that visualartifacts in still scenes are likely to noticeably degrade the viewingexperience more than visual artifacts in fast-motion scenes. Further,due to such perceptual effects, such fidelity metrics arecontent-dependent and, therefore, inconsistent across different types ofvideo data. For example, fidelity degradation in action movies thatconsist primarily of fast-motion scenes is less noticeable than fidelitydegradation in slow-paced documentaries.

As the foregoing illustrates, what is needed in the art are moreeffective techniques for predicting the perceived quality of videos.

SUMMARY OF THE INVENTION

One embodiment of the present invention sets forth acomputer-implemented method for estimating perceptual video quality. Themethod includes selecting a set of objective metrics that represent aplurality of deterministic video characteristics; for each trainingvideo included in a set of training videos, receiving a data set thatdescribes the training video, where the data set includes a subjectivevalue for a perceptual video quality metric and a set of objectivevalues for the set of objective metrics; from the data sets, deriving acomposite relationship that determines a value for the perceptual videoquality metric based on a set of values for the set of objectivemetrics; for a target video, calculating a first set of values for theset of objective metrics; and applying the composite relationship to thefirst set of values to generate an output value for the perceptual videoquality metric.

One advantage of the disclosed techniques for estimating perceptualvideo quality is that the composite relationship that defines theperceptual video quality metric fuses objective metrics based on direct,human observations. More specifically, because human feedback for a setof training videos guides the contribution of each of the objectivemetrics, applying the composite relationship to target videosgeneralizes human feedback. Consequently, the perceptual video qualitymetric reliably predicts perceived video quality. By contrast,conventional quality metrics typically measure signal fidelity—acharacteristic that does not necessarily track video quality asperceived by human vision systems.

BRIEF DESCRIPTION OF THE DRAWINGS

So that the manner in which the above recited features of the presentinvention can be understood in detail, a more particular description ofthe invention, briefly summarized above, may be had by reference toembodiments, some of which are illustrated in the appended drawings. Itis to be noted, however, that the appended drawings illustrate onlytypical embodiments of this invention and are therefore not to beconsidered limiting of its scope, for the invention may admit to otherequally effective embodiments.

FIG. 1 is a conceptual illustration of a system configured to implementone or more aspects of the present invention;

FIG. 2 is a block diagram illustrating the objective metric generationsubsystem and the perceptual quality trainer of FIG. 1, according to oneembodiment of the present invention;

FIG. 3 is a block diagram illustrating the objective metric generationsubsystem and the perceptual quality calculator of FIG. 1, according toone embodiment of the present invention;

FIG. 4 is a flow diagram of method steps for predicting perceptualvisual quality, according to one embodiment of the present invention;and

FIG. 5 is a flow diagram of method steps for calculating values for aperceptual visual quality score based on an empirically trained model,according to one embodiment of the present invention.

DETAILED DESCRIPTION

In the following description, numerous specific details are set forth toprovide a more thorough understanding of the present invention. However,it will be apparent to one of skilled in the art that the presentinvention may be practiced without one or more of these specificdetails.

System Overview

FIG. 1 is a conceptual illustration of a system 100 configured toimplement one or more aspects of the present invention. As shown, thesystem 100 includes a virtual private cloud (i.e., encapsulated sharedresources, software, data, etc.) 102 connected to a variety of devicescapable of transmitting input data and/or displaying video. Such devicesinclude, without limitation, a desktop computer 102, a smartphone 104,and a laptop 106. In alternate embodiments, the system 100 may includeany number and/or type of input, output, and/or input/output devices inany combination.

The virtual private cloud (VPC) 100 includes, without limitation, anynumber and type of compute instances 110. The VPC 100 receives inputuser information from an input device (e.g., the laptop 106), one ormore computer instances 110 operate on the user information, and the VPC100 transmits processed information to the user. The VPC 100 conveysoutput information to the user via display capabilities of any number ofdevices, such as a conventional cathode ray tube, liquid crystaldisplay, light-emitting diode, or the like.

In alternate embodiments, the VPC 100 may be replaced with any type ofcloud computing environment, such as a public or a hybrid cloud. Inother embodiments, the system 100 may include any distributed computersystem instead of the VPC 100. In yet other embodiments, the system 100does not include the VPC 100 and, instead, the system 100 includes asingle computing unit that implements multiple processing units (e.g.,central processing units and/or graphical processing units in anycombination).

As shown for the compute instance 110 ₀, each compute instance 110includes a central processing unit (CPU) 112, a graphics processing unit(GPU) 114, and a memory 116. In operation, the CPU 112 is the masterprocessor of the compute instance 110, controlling and coordinatingoperations of other components included in the compute instance 110. Inparticular, the CPU 112 issues commands that control the operation ofthe GPU 114. The GPU 114 incorporates circuitry optimized for graphicsand video processing, including, for example, video output circuitry. Invarious embodiments, GPU 114 may be integrated with one or more of otherelements of the compute instance 110. The memory 116 stores content,such as software applications and data, for use by the CPU 112 and theGPU 114 of the compute instance 110.

In general, the compute instances 110 included in the VPC 100 areconfigured to implement one or more applications. As shown, computeinstances 110 ₁-110 _(N) are configured as an encoder 120. The encoder120 implements any type of data compression techniques as known in theart and in any technically feasible fashion. In some embodiments, theencoder 140 is a parallel chunk encoder that partitions the source datainto multiple chunks and then performs data compression techniquesconcurrently on the chunks.

To comply with resource constraints, such as encoded data sizelimitations and available streaming bandwidth, the encoder 120implements lossy data compression techniques that eliminate selectedinformation. By eliminating information, the encoder 120 creates“compression” artifacts that introduce distortions when the source datais reconstructed. The visual quality of the reconstructed source data isoften further compromised by other elements included in the transcodingpipeline (i.e., the applications that translate the source data in oneformat to the reconstructed data in another format). For example,“scaling” artifacts may be introduced during the process of down-scalingand encoding the source data and then up-scaling the decoded data to thesource resolution at the display device.

To ensure an acceptable viewing experience, the quality of thereconstructed data and, indirectly, the caliber of the elements includedin the transcoding pipeline are typically evaluated at various points inthe design and delivery process using quality metrics. The values forthe quality metrics are then used to guide the development ofapplications (e.g., encoders) and the real-time optimization of contentdelivery, such as stream-switching algorithms that are quality-aware.

Many widely applied quality metrics (e.g., mean-squared-error (MSE) andpeak signal-to-noise ratio (PSRN)) measure fidelity—the faithfulness ofthe reconstructed data to the source data. However, fidelitymeasurements do not reflect psycho-visual phenomena affecting the humanvisual system (HVS) such as masking, contrast sensitivity, or the highlystructured content in natural images. Further, due to such imperfectlyreflected perceptual effects, such fidelity metrics arecontent-dependent—the values are not comparable across different typesof video data. For instance, video with grain noise is relativelyheavily penalized in PSNR although the visual impact detectable by humanviewers is relatively low. In general, conventional quality metrics arenot a reliable indication of the visual quality as perceived by humansand, therefore, the acceptability of the viewing experience.

For this reason, one or more of the compute instances 110 in the VPC 102implement machine learning techniques to institute a consistentperceptual quality metric. Notably, a perceptual quality score 165(i.e., value for the perceptual quality metric) correlates in auniversal manner to subjective human visual experience irrespective ofthe type of video content. Any type of learning algorithm as known inthe art may be leveraged to implement the consistent perceptual qualitymetric. In some embodiments, a support vector machine (SVM) provides theframework for the consistent perceptual quality metric. In otherembodiments, a neural network implements the algorithms to establish theconsistent perceptual quality metric.

In a training phase, depicted in FIG. 1 with dotted lines, a perceptualquality trainer 150 creates a perceptual quality model 155. Theperceptual quality model 155 is a supervised learning model thatcombines objective metrics 145 to optimally track the values for thesubjective metric 135 assigned during viewings of training data. Theobjective metric subsystem 140 generates the objective metrics 145 basedon comparison operations between the training data and the correspondingencoded training data. Such objective metrics 145 are referred to asfull-reference quality indices, and may be generated in any technicallyfeasible fashion. After a decoder 125 generates reconstructed trainingdata from the encoded training data, viewers 110 watch the reconstructeddata on display devices, such as the screen of the laptop 106, andpersonally rate the visual quality—assigning values to the subjectivemetric 135.

The perceptual quality trainer 150 receives the calculated values forthe objective metrics 145 and the human-assigned values for thesubjective metric 135. The perceptual quality trainer 150 then trainsthe perceptual quality model 155 based on these metrics. Morespecifically, the perceptual quality trainer 150 executes learningalgorithms that recognize patterns between the objective metrics 145 andthe subjective metric 135. Subsequently, the perceptual quality trainer150 configures the perceptual quality model 155 to fuse values for theobjective metrics 145 into a perceptual quality score 165 that reflectsthe value for the subjective metric 135 and, consequently, theexperience of the viewers 110.

In a scoring phase, depicted in FIG. 1 with solid lines, a perceptualquality calculator 160 receives the perceptual quality model 155 and thevalues for the objective metrics 145 for target data. The perceptualquality calculator 160 applies the perceptual quality model 155 to thevalues for the objective metrics 145 and generates the perceptualquality score 165 for the target data. The values for the objectivemetrics 145 may be generated in any technically feasible fashion. Forexample, the objective metric subsystem 140 may compare any referencedata (e.g., source data) to any derived target data (e.g., encodedsource data) to calculate the values for the objective metrics 145.

Training Phase

FIG. 2 is a block diagram illustrating the objective metric generationsubsystem 140 and the perceptual quality trainer 150 of FIG. 1,according to one embodiment of the present invention. The objectivemetric generation subsystem 140 may be implemented in any technicallyfeasible fashion and may include any number of separate applicationsthat each generates any number of values for the objective metrics 145.The perceptual quality trainer 150 includes, without limitation, asupport vector machine (SVM) model generator 240 and a temporaladjustment identifier 250.

Upon receiving training data 205 and encoded training data 295 for a setof training videos, the objective metric generation subsystem 140computes the values for the objective metrics 145. The training videosmay include any number and length of video clips that represent therange of video types to be represented by the perceptual quality score165. For example, in one embodiment the video clips in the training setspan a diverse range of high level features (e.g., animation, sports,indoor, camera motion, face close-up, people, water, obvious salience,object number) and low level characteristics (e.g. film grain noise,brightness, contrast, texture, motion, color variance, color richness,sharpness).

In some embodiments the set of training videos is the MCL-V videodatabase of video clips that is available publically from the Universityof Southern California. In other embodiments, the ML-V video database ofvideo clips is supplemented with selected high film grain clips andanimation titles to increase the diversity and the robustness of the setof training videos. The training data 205 includes the training videosand the encoded training data 295 is derived from the training data 205.More specifically, for each of the clips included in the training data205, the encoder 150 is configured to encode the clip repeatedly, at avariety of different resolutions and/or quality levels (i.e., bitrates).In this fashion, a predetermined number of encoded clips are generatedfrom each video clip in the training set and these encoded clips formthe encoded training data 295.

In general, each video quality metric exhibits both strengths andweaknesses. To leverage the strengths and mitigate the weaknesses, theobjective metric generation subsystem 140 is configured to calculate aset of the objective metrics 145 that, together, provide valuableinsight into the visual quality across the range of the encoded trainingdata 295. The selection of the objective metrics 145 may be made in anytechnically feasible fashion to address any number of anticipatedartifacts. For instance, in some embodiments, the objective metrics 145are empirically selected to assess degradation caused by compression(i.e., blockiness) and scaling (i.e. blurriness).

As shown, the objective metrics 145 include a detail loss measure (DLM)242, a visual information fidelity (VIF) 244, and an anti-noisesignal-to-noise ratio (ANSNR) 246. The DLM 242 is based on applyingwavelet decomposition to identify the blurriness component of signals.The DLM 242 is relatively good at detecting blurriness in intermediatequality ranges, but is relatively poor at discriminating quality inhigher quality ranges. The VIF 244 is based on applying a wavelettransformation to analyze signals in the frequency domain. The VIF 244is relatively good at detecting slight bluing artifacts, but is relativepoor at detecting blocking artifacts.

The ANSNR 246 is designed to mitigate some drawbacks of SNR for filmcontent. Prior to performing the SNR calculation, the objective metricgeneration subsystem 140 applies a weak low-pass filter to the trainingdata 205 and a stronger low-pass filter to the encoded training data295. The ANSNR 246 is relatively fast to compute and good for detectingcompression artifacts and strong scaling artifacts. However, the ANSNR246 ignores slight blurring artifacts and, consequently, is notsensitive to minor quality changes in the high quality ranges.

As a further optimization, since the human visual system is lesssensitive to degradation during periods of high motion, the objectivemetric generation subsystem 140 computes motion values 248. For eachframe, the object metric generation subsystem 140 computes the motionvalue 248 as the mean co-located pixel difference of the frame withrespect to the previous frame. Notably, to reduce the likelihood thatnoise is misinterpreted as motion, the object metric generationsubsystem 140 applies a low-pass filter before performing the differencecalculation.

The values for the subjective metric 135 are assigned by the viewers 110after watching the training data 205 and decoded versions of the encodedtraining data 295, referred to herein as reconstructed training data, onany number and type of display devices. In one embodiment, each of theviewers 110 watch each training clip side-by-side with each of thereconstructed training clips and assigns values to the subjective metric135. The value for the subjective metric 135 is an absolute value thatindicates the perceived visual quality. For instance, in one embodiment,the value for the subjective metric 135 may vary from 0 through 100. Ascore of 100 indicates that the reconstructed training clip appearsidentical to the training clip. A score below 20 indicates that thereconstructed training clip loses significant scene structure andexhibits considerable blurring relative to the training clip.

Subsequently, the SVM model generator 240 receives the motion values248, values for the objective metrics 145, and values for the subjectivemetric 135 for the encoded training data 295. The SVM model generator240 then applies learning algorithms to train the perceptual qualitymodel 150. For the encoded training data 295, the SMV model generator240 identifies correlations between the observed values for thesubjective metric 135 and the calculated values for the objectivemetrics 145 as well as the motion values 248. The SVM model generator240 then generates the perceptual quality model 155—a fusion of theobjective metrics 135 and the motion value 248 that estimates thesubjective metric 135. As persons skilled in the art will recognize, theSVM model generator 240 may implement any of a number of learningalgorithms to generate any type of model. In alternate embodiments, theSVM model generator 240 may be replaced with any processing unit thatimplements any type of learning algorithm, such as a neural network.

The temporal adjustment identifier 250 is configured to tune theperceptual quality model 155 for corner cases. Notably, for very highmotion scenes (i.e., high motion values 248), the perceptual qualitymodel 155 may not adequately represent temporal masking effects.Consequently, the temporal adjustment identifier 250 generates atemporal adjustment 255 that is applied to the perceptual quality model155 for such scenes. In some embodiments, the temporal adjustment 255includes a threshold and a percentage. The temporal adjustment 255 isapplied in conjunction with the perceptual quality model 155, increasingthe perceptual quality score 165 computed via the perceptual qualitymodel 155 by the percentage.

Scoring Phase

FIG. 3 is a block diagram illustrating the objective metric generationsubsystem 140 and the perceptual quality calculator 160 of FIG. 1,according to one embodiment of the present invention. As shown, theperceptual quality calculator 150 includes, without limitation, asupport vector machine (SVM) mapper 360 and a temporal adjuster 370. Theperceptual quality calculator 150 operates during the scoringphase—computing perceptual quality scores 165 for the encoded data 195that is derived from the source data 105 based on the “trained”perceptual quality model 155 and the temporal adjustment 255.

The SVM mapper 360 may be configured with any number of perceptualquality models 155 and temporal adjustments 255 that correspond to anynumber of training data 105. In some embodiments, a model selectionmodule (not shown) classifies training data 105 of similar content intogroups and then assigns the perceptual quality model 155 based on thecontent of the encoded data 195 to be assessed. For example, one set oftraining data 105 may include relatively high quality videos and,therefore, the corresponding perceptual quality model 155 is optimizedto determine the perceptual quality score 165 for high quality encodeddata 195. By contrast, another set of training data 105 may includerelatively low quality videos and, therefore, the correspondingperceptual quality model 155 is optimized to determine the perceptualquality score 165 for low quality encoded data 195.

Upon receiving the source data 105 and the encoded data 195 derived fromthe source data 105, the objective metric generation subsystem 140computes the values for the objective metrics 145 and the motion values248. In general, the values for the objective metrics 145 and the motionvalues 248 may be determined in any technically feasible fashion. Forinstance, some embodiments include multiple objective metriccalculators, and each objective metric calculator configures a differentobjective metric.

The SVM mapper 360 applies the perceptual quality model 155 to theobjective metrics 145 and the motion values 248 to generate a perceptualquality score 165. Subsequently, the temporal adjuster 370 selectivelyapplies the temporal adjustment 255 to the perceptual quality score 165to fine-tune corner cases. In one embodiment, the temporal adjuster 370compares the motion values 240 to a threshold included in the temporaladjustment 255. If the motion value 240 exceeds the threshold, then thetemporal adjuster 370 increases the perpetual quality score 165 by apercentage included in the temporal adjustment 255 to reflect theinherent pessimism of the perceptual quality model 155 for high motionscenes. Because the perceptual quality model 155 and the temporaladjustment 255 track quality observed by the viewers 110, the perceptualquality score 165 reflects the quality of the encoded data 185 whenviewed by humans.

Note that the techniques described herein are illustrative rather thanrestrictive, and may be altered without departing from the broaderspirit and scope of the invention. In particular, the perceptual qualitytrainer 150 may be replaced with any module that implements any numberof machine learning processes to generate a model that fuses multipleobjectively calculated values to track an experimentally observed visualquality. Correspondingly, the perceptual quality calculator 160 may bereplaced with any module that applies the model in a consistent fashion.Further, the perceptual quality trainer 150 may include any number ofadjustment identification modules designed to fine-tune the generatedmodel, and the perceptual quality calculator 160 may include any numberof adjustment calculators that apply the identified adjustments.

The granularity (e.g., per frame, per scene, per shot, per 6 minuteclip, etc.) of the training data 105, the objective metrics 145, thesubjective metrics 135, and the motion values 245 may be vary within andbetween implementations. As persons skilled in the art will recognize,conventional mathematical techniques (e.g., averaging, extrapolating,interpolating, maximizing, etc.) may be applied to the objective metrics145, the subjective metrics 135, and/or the motion values 245 in anycombination to ensure measurement unit consistency. Further, theperceptual quality trainer 150 and the perceptual quality calculator 160may be configured to determine the perceptual quality model 155, thetemporal adjustment 255, and/or the perceptual quality score 160 at anygranularity.

Predicting Human-Perceived Quality

FIG. 4 is a flow diagram of method steps for predicting perceptualvisual quality, according to one embodiment of the present invention.Although the method steps are described with reference to the systems ofFIGS. 1-3, persons skilled in the art will understand that any systemconfigured to implement the method steps, in any order, falls within thescope of the present invention.

As shown, a method 400 begins at step 404, where the perceptual qualitytrainer 150 receives the training data 205. The training data 205 mayinclude any number and length of video clips. For example, in oneembodiment the training data 205 includes sixteen six minute clips. Atstep 406, the encoder 120 derives the encoded test data 295 from thetraining data 205 for any number of resolutions and combination of bitrates. In general, the resolutions and bit rates are selected to reflecttarget supported ranges for viewing devices and/or streaming bandwidth.

At step 406, the perceptual quality trainer 150 receives values for thesubjective metric 135 for reconstructed video clips ((i.e., decoded,scaled, etc.) derived from the encoded training data 295. The perceptualquality trainer 150 may obtain values for the subjective metric 135 inany form and may perform any number of post-processing operations (e.g.,averaging, removal of outlying data points, etc.). In alternateembodiments, the perceptual quality trainer 150 may receive and processdata corresponding to any number of subjective metrics 135 in anytechnically feasible fashion.

For example, in some embodiments, the perceptual quality trainer 150receives feedback generated during a series of side-by-side, human(e.g., by the viewers 100) comparisons of the training data 205 and thereconstructed video clips (i.e., decoded, scaled, etc.) derived from theencoded training data 295. For each of the reconstructed video clips,the feedback includes a value for the subjective metric 135 for thecorresponding encoded test data 295. The value of the subjective metric135 reflects the average observed visual quality based on an absolute,predetermined, quality scale (e.g., 0-100, where 100 represents nonoticeable artifacts).

At step 410, the objective metric generation subsystem 140 computesvalues for the objective metrics 145 for the encoded test data 295 basedon both the encoded test data 295 and the training data 205. Theobjective metric generation subsystem 140 may select the objectivemetrics 145 and then compute the values for the objective metrics 145 inany technically feasible fashion. For example, in some embodiments theobjective metric generation subsystem 140 is configured to computevalues for the detail loss measure (DLM) 242, the visual informationfidelity (VIF) 244, and the anti-noise signal-to-noise ratio (ANSNR)246.

As part of step 410, the objective metric generation subsystem 140 mayalso compute any other type of spatial or temporal data associated withthe encoded test data 295. In particular, the objective metricgeneration subsystem 140 calculates the motion values 248 for each frameincluded in the encoded test data 295—the temporal visual difference.

At step 412, the support vector machine (SVM) model generator 240performs machine learning operations—training the perceptual qualitymodel 155 to track the values for the subjective metric 135 based on afusion of the values for the objective metrics 145 and the motion values248. At step 414, the perceptual quality trainer 150 determines whetherthe perceptual quality model 155 accurately tracks the values for thesubjective metric 135 during periods of high motion, If, at step 414,the perceptual quality trainer 150 determines that the accuracy of theperceptual quality model 155 is acceptable, then this method proceedsdirectly to step 418.

If, at step 414, the perceptual quality trainer 150 determines that theaccuracy of the perceptual quality model 155 is unacceptable, then thismethod proceeds to step 416. At step 416, the temporal adjustmentidentifier 250 determines a threshold beyond which the perceptualquality score 165 computed based on the perceptual quality model 155 isunacceptably pessimistic. The temporal adjustment identifier 250 alsodetermines a percentage increase that, when applied to the perceptualquality score 165 computed based on the perceptual quality model 155,improves the accuracy of the perceptual quality score 165. Together, thethreshold and the percentage increase form the temporal adjustment 255.

At step 418, the perceptual quality calculator 160 calculates theperceptual quality scores 165 for the encoded data 195 based on theperceptual quality model 165 and, when present, the temporal adjustment255. In general, the perceptual quality calculator 160 computes theperceptual quality score 165 by applying the perceptual quality model155 to the values for the objective metrics 155 and the motion values248 for the encoded data 195 in any technically feasible fashion.

For example, in some embodiments, the perceptual quality calculator 150performs the method steps outlined below in conjunction with FIG.5—leveraging the trained perceptual quality model 155 to obtainperceptual quality scores 165 (i.e., values of the subjective metric135). Notably, during the training phase the perceptual quality model165 directly incorporates human feedback for the training data 205.Subsequently, during the scoring phase the trained perceptual qualitymodel 165 enables the generalization of this human feedback to anynumber and type of source data 105.

FIG. 5 is a flow diagram of method steps for calculating values for aperceptual visual quality score based on an empirically trained model,according to one embodiment of the present invention. Although themethod steps are described with reference to the systems of FIGS. 1-3,persons skilled in the art will understand that any system configured toimplement the method steps, in any order, falls within the scope of thepresent invention.

As shown, a method 500 begins at step 516, where the perceptual qualitycalculator 160 receives the perceptual quality model 155 and thetemporal adjustment 255. In alternate embodiments, the temporaladjustment 255 may be omitted. In other embodiments, the temporaladjustment 255 is replaced with any number of other adjustments that aredesigned to fine-tune the perceptual quality score 165. The perceptualquality model 155 may be generated in any technically feasible fashion.For example, in some embodiments, the perceptual quality trainer 140performs the method steps 406-416 outlined in FIG. 4.

At step 518, the perceptual quality calculator 160 receives the sourcedata 105. At step 520, the encoder 120 derives the encoded data 195 fromthe source data 205 for a target resolution and/or bit rate. At step522, the objective metric generation subsystem 140 computes values forthe objective metrics 145 for the encoded data 195 based on the encodeddata 195 and, for optionally, the source data 105. The objective metricgeneration subsystem 140 also computes the motion values 248 for eachframe of the encoded data 195. In general, the perceptual qualitycalculator 160 is configured to calculator the values for theindependent variables in the perceptual quality model 155.

At step 524, the support vector machine (SVM) mapper 360 applies theperceptual quality model 155 to the values for the objective metrics 145and the motion values 248 for the encoded data 195 to generate theperceptual quality score 165. At step 526, the temporal adjuster 370determines whether the motion values 248 of one or more frames exceedthe threshold specified in the temporal adjustment 255. If, at step 526,the temporal adjuster 370 determines that none of the motion values 248exceed the threshold, then the perceptual quality calculator 160considers the perceptual quality score 165 to accurately predict theexpected viewing experience and the method 500 ends.

If, at step 526, the temporal adjuster 370 determines that any of themotion values 248 exceed the threshold, then the temporal adjuster 370considers the frames to reflect a period of high motion, and the method500 proceeds to step 526. At step 526, the temporal adjuster 370increases the perceptual quality score 165 by a threshold percentage(specified in the temporal adjustment 255) to compensate for thepessimism of the perceptual quality model 155 during periods of highmotion, and the method 500 ends.

In sum, the disclosed techniques may be used to efficiently and reliablypredict perceptual video quality. A perceptual quality trainerimplements a support vector machine (SVM) to generate a perceptualquality model. Notably, for a training set of videos, the SVM isconfigured to fuse values for a set of objective metrics and temporalmotion into a perceptual quality score—a subjective visual quality scorethat is based on human video-viewing feedback. Subsequently, aperceptual quality calculator applies the perceptual quality model tovalues for the objective metrics and temporal motion for target videosto generate corresponding values for the perceptual quality metric(i.e., visual quality score).

Advantageously, training the perceptual quality model using directobservations made by human visual systems enables the perceptual qualitycalculator to efficiently calculate quality scores that reliably predictperceived video quality in an absolute manner. By contrast, conventionalquality metrics typically measure signal fidelity—a content-dependent,inconsistent, and unreliable indication of real world viewingappreciation. Further, by separating the initial empirically-basedtraining phase from the subsequent per-video deterministic calculationphase, the disclosed techniques are expeditious and scalable.Consequently the perceptual quality model both reduces the time requiredto develop and accurately evaluate encoders and enables time-sensitiveencoding applications, such as real-time quality-aware stream-switching.

The descriptions of the various embodiments have been presented forpurposes of illustration, but are not intended to be exhaustive orlimited to the embodiments disclosed. Many modifications and variationswill be apparent to those of ordinary skill in the art without departingfrom the scope and spirit of the described embodiments.

Aspects of the present embodiments may be embodied as a system, methodor computer program product. Accordingly, aspects of the presentdisclosure may take the form of an entirely hardware embodiment, anentirely software embodiment (including firmware, resident software,micro-code, etc.) or an embodiment combining software and hardwareaspects that may all generally be referred to herein as a “circuit,”“module” or “system.” Furthermore, aspects of the present disclosure maytake the form of a computer program product embodied in one or morecomputer readable medium(s) having computer readable program codeembodied thereon.

Any combination of one or more computer readable medium(s) may beutilized. The computer readable medium may be a computer readable signalmedium or a computer readable storage medium. A computer readablestorage medium may be, for example, but not limited to, an electronic,magnetic, optical, electromagnetic, infrared, or semiconductor system,apparatus, or device, or any suitable combination of the foregoing. Morespecific examples (a non-exhaustive list) of the computer readablestorage medium would include the following: an electrical connectionhaving one or more wires, a portable computer diskette, a hard disk, arandom access memory (RAM), a read-only memory (ROM), an erasableprogrammable read-only memory (EPROM or Flash memory), an optical fiber,a portable compact disc read-only memory (CD-ROM), an optical storagedevice, a magnetic storage device, or any suitable combination of theforegoing. In the context of this document, a computer readable storagemedium may be any tangible medium that can contain, or store a programfor use by or in connection with an instruction execution system,apparatus, or device.

Aspects of the present disclosure are described above with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems) and computer program products according to embodiments of thedisclosure. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer program instructions. These computer program instructions maybe provided to a processor of a general purpose computer, specialpurpose computer, or other programmable data processing apparatus toproduce a machine, such that the instructions, which execute via theprocessor of the computer or other programmable data processingapparatus, enable the implementation of the functions/acts specified inthe flowchart and/or block diagram block or blocks. Such processors maybe, without limitation, general purpose processors, special-purposeprocessors, application-specific processors, or field-programmable

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods and computer program products according to variousembodiments of the present disclosure. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof code, which comprises one or more executable instructions forimplementing the specified logical function(s). It should also be notedthat, in some alternative implementations, the functions noted in theblock may occur out of the order noted in the figures. For example, twoblocks shown in succession may, in fact, be executed substantiallyconcurrently, or the blocks may sometimes be executed in the reverseorder, depending upon the functionality involved. It will also be notedthat each block of the block diagrams and/or flowchart illustration, andcombinations of blocks in the block diagrams and/or flowchartillustration, can be implemented by special purpose hardware-basedsystems that perform the specified functions or acts, or combinations ofspecial purpose hardware and computer instructions.

While the preceding is directed to embodiments of the presentdisclosure, other and further embodiments of the disclosure may bedevised without departing from the basic scope thereof, and the scopethereof is determined by the claims that follow.

What is claimed is:
 1. A computer-implemented method for estimatingperceptual video quality, the method comprising: selecting a set ofobjective metrics that represent a plurality of deterministic videocharacteristics; for each training video included in a set of trainingvideos, receiving a subjective value for a perceptual video qualitymetric and a set of objective values for the set of objective metrics,wherein the subjective value and the set of objective values describethe training video; deriving a composite relationship based on acorrelation between the subjective value, the set of objective values,and a measure of pixel motion within at least one of the set of trainingvideos, wherein the composite relationship specifies a level ofcontribution for at least one of the set of objective metrics to theperceptual video quality metric; for a target video, calculating a firstset of values for the set of objective metrics; and applying thecomposite relationship to the first set of values to generate an outputvalue for the perceptual video quality metric.
 2. Thecomputer-implemented method of claim 1, wherein deriving the compositerelationship comprises performing one or more training operations on thedata sets.
 3. The computer-implemented method of claim 2, whereinperforming one or more training operations on a given data set comprisesapplying a support vector machine algorithm or an artificial neuralnetwork algorithm to the set of objective values included in the dataset.
 4. The computer-implemented method of claim 1, further comprising:determining that a value included in the first set of values exceeds apredetermined threshold; and modifying the output value for theperceptual quality metric based on an adjustment factor.
 5. Thecomputer-implemented method of claim 1, further comprising: computing amotion value based on pixel differences between two consecutive framesof the target video; determining that the motion value exceeds apredetermined threshold; and increasing the output value for theperceptual quality metric by a predetermined amount.
 6. Thecomputer-implemented method of claim 1, wherein the set of objectivemetrics includes at least one of detail loss measure and visualinformation fidelity.
 7. The computer-implemented method of claim 1,wherein the set of objective metrics includes an anti-noisesignal-to-noise ratio, the target video is derived from a source video,and calculating a first value for the anti-noise signal-to-noise ratiocomprises: applying a first low pass filter to the source video;applying a second low pass filter to the target video that is strongerthan the first low pass filter; and performing one or moresignal-to-noise ratio calculations based on the filtered source videoand the filtered target video.
 8. The computer-implemented method ofclaim 1, wherein a first training video included in the set of trainingvideos includes at least one of compressed data and scaled data.
 9. Thecomputer-implemented method of claim 1, wherein a first subjective valuefor the perceptual video quality metric is a human-observed score forthe visual quality of a reconstructed video that is derived from thefirst training video.
 10. A non-transitory computer-readable storagemedium including instructions that, when executed by a processing unit,cause the processing unit to estimate perceptual video quality byperforming the steps of: selecting a set of objective metrics thatrepresent a plurality of deterministic video characteristics; for eachtraining video included in a set of training videos, receiving asubjective value for a perceptual video quality metric and a set ofobjective values for the set of objective metrics, wherein thesubjective value and the set of objective values describe the trainingvideo; deriving a composite relationship based on a correlation betweenthe subjective value, the set of objective values, and a measure ofpixel motion within at least one of the set of training videos, whereinthe composite relationship specifies a level of contribution for atleast one of the set of objective metrics to the perceptual videoquality metric; for a target video, calculating a first set of valuesfor the set of objective metrics; and applying the compositerelationship to the first set of values to generate an output value forthe perceptual video quality metric.
 11. The non-transitorycomputer-readable storage medium of claim 10, wherein deriving thecomposite relationship comprises performing one or more trainingoperations on the data sets.
 12. The non-transitory computer-readablestorage medium of claim 10, further comprising: computing a motion valuebased on pixel differences between two consecutive frames of the targetvideo; determining that the motion value exceeds a predeterminedthreshold; and increasing the output value for the perceptual qualitymetric by a predetermined amount.
 13. The non-transitorycomputer-readable storage medium of claim 10, wherein a first trainingvideo included in the set of training videos includes compressed dataderived from a first original video.
 14. The non-transitorycomputer-readable storage medium of claim 13, wherein a first subjectivevalue for the perceptual video quality metric indicates the variationbetween a visual quality of the first original video and a visualquality of a reconstructed training video that is derived from the firsttraining video based on one or more decompression operations.
 15. Thenon-transitory computer-implemented method of claim 13, wherein a firstsubjective value for the perceptual video quality metric is ahuman-observed score for the visual quality of a video that is derivedfrom the first training video based on one or more decompressionoperations.
 16. The non-transitory computer-implemented method of claim1, wherein the set of objective metrics includes an anti-noisesignal-to-noise ratio, the target video is derived from a source video,and calculating a first value for the anti-noise signal-to-noise ratiocomprises: applying a first low pass filter to the source video;applying a second low pass filter to the target video that is strongerthan the first low pass filter; and performing one or moresignal-to-noise ratio calculations based on the filtered source videoand the filtered target video.
 17. The non-transitory computer-readablestorage medium of claim 10, wherein the composite relationship is anequation.
 18. The non-transitory computer-readable storage medium ofclaim 17, wherein applying the composite relationship to the first setof values comprises solving the equation for the values included in thefirst set of values.
 19. A system configured to estimate perceptualvideo quality based on a set of objective metrics that represent aplurality of deterministic video characteristics, the system comprising:an encoder configured to generate a set of training videos from aplurality of original videos; a perceptual quality trainer configuredto: for each training video included in a set of training videos,receive a subjective value for a perceptual video quality metric and aset of objective values for the set of objective metrics, wherein thesubjective value and the set of objective values describe the trainingvideo; deriving a composite relationship based on a correlation betweenthe subjective value, the set of objective values, and a measure ofpixel motion within at least one of the set of training videos, whereinthe composite relationship specifies a level of contribution for atleast one of the set of objective metrics to the perceptual videoquality metric; and a perceptual quality calculator configured to: for atarget video, calculate a first set of values for the set of objectivemetrics; and apply the composite relationship to the first set of valuesto generate an output value for the perceptual video quality metric. 20.The system of claim 19, wherein deriving the composite relationshipcomprises performing one or more training operations on the data sets.21. A computer-implemented method for estimating perceptual videoquality, the method comprising: for each training video included in aset of training videos, receiving a data set that describes the trainingvideo, wherein the data set includes a subjective value for a perceptualvideo quality metric, a measure of pixel motion within the trainingvideo, and a set of objective values for a set of objective metrics thatincludes an anti-noise signal-to-noise ratio, a detail loss measure, anda visual information fidelity measure; from the data sets, deriving acomposite relationship based on a correlation between the subjectivevalue, the set of objective values, and the measure of pixel motion; fora target video, calculating a first set of values for the motion and afirst set of values for the set of objective metrics; applying thecomposite relationship to the first set of values for the motion and thefirst set of values for the set of objective metrics to generate anoutput value for the perceptual video quality metric; determining that afirst motion value included in the first set of values for the motionexceeds a predetermined threshold; and modifying the output value forthe perceptual quality metric based on an adjustment factor that isassociated with the motion.